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Abstract 

For decades, the DoD has employed numerous reporting and monitoring tools for characterizing the acquisition cost 
of its major programs. These tools have resulted in dozens of studies thoroughly documenting the magnitude and 
extent of DoD acquisition cost growth. Curiously, though, there have been extremely few studies regarding the 
behavior of the other cost component of a system’s life cycle: Operating and Support (O&S) costs. This is 
particularly strange considering that O&S costs tend to dominate the total life cycle cost (LCC) of a program, and that 
LCCs are widely regarded as the preferred metric for assessing actual program value. The upshot for not examining 
such costs is that the DoD has little knowledge of how LCC estimates behave over time, and virtually no insights 
regarding their accuracy. In recent years, however, enough quality LCC data has amassed to conduct a study to 
address these deficiencies. This paper describes a method for conducting such a study, and represents (to the authors’ 
knowledge) the first broad-based attempt to do so. The results not only promise insights into the nature of current 
LCC estimates, but also suggest the possibility of improving the accuracy of DoD LCC estimates via a stochastically- 
based model. 

©2012 Published by Elsevier Ltd. Selection 
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1. Introduction and Motivation 

The Department of Defense (DoD) has long stressed the importance of monitoring and reducing the 
total life cycle cost (also sometimes referred to as “total ownership cost”) of its systems. Consider the 
following excerpts from the venerable DOD Directive 5000.01: Defense Acquisition System [1] — 

• Programs shall be managed through the application of a systems engineering approach that optimizes 
total system performance and minimizes total ownership costs. 
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• Planning for ... the estimation of total ownership costs shall begin as early as possible. 

• To the greatest extent possible, the MDAs [Milestone Decision Authorities] shall identify the total 

costs of ownership, and at a minimum, the major drivers of total ownership costs. 

Similar guidance can be found in a number of other long-standing authoritative sources (e.g., DoDI 
5000.02, and DoDM 5000.04 [2,3]), as well as the recently enacted Weapon Systems Acquisition Reform 
Act (WSARA) of 2009 [4], the importance of which will be discussed later. 

The DoD’s definition of Life Cycle Cost (LCC) is the total cost to the government spanning all phases 
of the program’s life: development, procurement, operation, sustainment, and disposal [3]. Note that this 
definition includes some costs accrued before a system formally enters the acquisition phase (e.g., 
requirements definition and concept development) as well as certain costs accrued as the system 
transitions out of sustainment (e.g., demilitarization and disposal). These initial and final costs—though 
sometimes sizeable from an absolute perspective—are almost always negligible when compared to the 
costs incurred during the program’s acquisition phase and its Operations and Support (O&S) phase. 
Consequently, one can state, to a high degree of accuracy, that a system’s LCC is simply the sum of its 
total acquisition costs and its total O&S costs. 

Of these two cost components, the DoD has historically placed significantly greater emphasis on the 
acquisition side of the equation. Over the years, a plethora of control and oversight accountability 
mechanisms—from milestones and congressional reporting to baselines and breaches—have been 
implemented with the expressed purpose of improving the execution and/or management of the 
acquisition phase of defense programs. Meanwhile, sustainability considerations have been perennially 
neglected or subordinated to acquisition requirements or program survival [4,5]. 

At first, this disproportionate emphasis on the acquisition phase might seem odd given the well-known 
fact that the majority of DoD system costs tend to be incurred during the O&S phase (recent estimates put 
the share of O&S costs at about 60-75 percent of the overall life cycle costs [4]). However, it does make 
sense both practically and strategically. From a practical perspective, it is much easier to implement the 
aforementioned control and oversight accountability mechanisms within the acquisition phase, with its 
relatively simple chain of command, tighter span of control, and shorter duration. And strategically 
speaking, focusing on the acquisition makes ample monetary sense: Though fewer dollars are expended 
during acquisition, the actions and decisions being made during this phase have a much greater impact on 
the life cycle cost than those being made during the O&S phase. This entire dynamic (which is really the 
consummate justification for systems engineering) is well depicted in the classic cost curves of Figure 1 
below [6]. 



Figure 1. Relationship between program phases and life cycle cost [6] 
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By virtue of its traditional focus on the acquisition component of a system’s life cycle, the DoD has 
managed to gain a variety of valuable insights into the nature of the acquisition costs of defense, including 
how accurate acquisition cost estimates are and how they tend to behave over time. These insights have 
provided the framework for many revisions to the acquisition process and provided the opportunity for 
numerous improvements to the acquisition cost component of a system’s LCC. Unfortunately, the same 
cannot be said for O&S cost projections. Despite an increased focus on O&S costs in recent years, the fact 
remains that the DoD simply still does not know how O&S cost estimates compare to reality. 
Consequently, DoD emphasis on a program’s life cycle cost is effectively a hollow requirement. Without 
knowledge of the validity of a program’s O&S cost estimates, we cannot have confidence in its LCC 
estimates. And without confidence in LCC estimates, any efforts to reduce LCCs are effectively nullified, 
and attempts to meaningfully discern the value of competing systems based on their respective LCCs are 
rendered futile. 

If one accepts the premise that accurate LCC estimates are of vital importance to DoD decision makers, 
then it is imperative that the behavior of O&S cost estimates be fully characterized. And the opportunity 
to do so has never been better. The combination of long-cultivated O&S reporting requirements and the 
fact that enough time has elapsed for the resultant data to sufficiently accumulate, allows analysts—for the 
first time in history—to conduct a relatively comprehensive assessment of DoD O&S cost behavior. The 
authors have accomplished such an assessment, and outline the employed methodology in this paper. 

2. Background 

To better understand the proposed methodology for characterizing current O&S cost estimates for DoD 
systems, it is necessary to be familiar with the relevant extant research and reporting mechanisms. 

2.1. Existing studies 

Even assuming an awareness that the acquisition phase is regarded as more important than the O&S 
phase, the existing number of studies seeking to characterize O&S costs for defense systems still seems 
shockingly sparse. Between 1945 and 2009, there were over 130 separate studies and commissions 
focused on the acquisition of DoD systems, dozens of which involved the nature of acquisition cost 
behavior [4]. During this same time period, there appears not to be a single published study pertaining to 
how system costs behave during the O&S phase. 

It would seem that WSARA has infused a greater sense of urgency into the DoD regarding the need to 
characterize O&S costs. Following its edict to “review existing systems and methods of the DoD for 
tracking and assessing O&S costs,” there has been a relative flurry of reports (four) quantitatively 
examining O&S costs for defense programs. Before summarizing these reports, however, a clarification 
regarding terminology is warranted. Within the acquisition community, “cost growth” is a well- 
established term with a specific meaning: the degree to which the actual costs of a system vary from the 
estimated (i.e., baseline) costs. When these O&S studies use the term “cost growth,” however, it has a 
broader meaning, denoting the difference between initial and final estimated costs in some cases, and the 
difference between initial and final actual costs in others. Although this conception of cost growth does 
allow us to gauge the stability and precision of the O&S cost estimates (or actuals), per se, it does not 
provide insight into the accuracy of DoD O&S cost estimates, which is the topic of this paper. 

The most comprehensive O&S study can be found in the WSARA Product Support Assessment [4]. In 
this study, 34 weapon systems are analyzed along several proposed dimensions of sustainment 
effectiveness. One new measure, dubbed “Sustained Cost Management,” analyzes the year-to-year 
changes in actual O&S costs, which may be an effective measure of O&S costs, but does not help us 
achieve the goal of ascertaining the accuracy or reliability of historical O&S cost estimates. 

Another broad study was conducted by the Center for Naval Analyses (CNA), also in 2009 [5], The 
study, which involved 26 Navy programs, had the expressed and promising purpose of vetting the 
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hypothesis that actual system O&S costs are “radically” exceeding early estimates. However, rather than 
comparing estimated costs to actual costs, the analysts decided—in all but three cases—to compare initial 
and final O&S cost estimates, choosing to assume that the final O&S cost estimate could reasonably 
represent actual costs 1 . Furthermore, although the O&S costs of three programs were analyzed by 
comparing estimates to actuals, unfortunately the numerical results were not provided in the report. 

In 2010, a narrower, but more in-depth study was performed by the Institute of Defense Analysis 
(IDA) which examined the Air Force C-17A program [7]. Like the majority of the systems in the CNA 
study, the C-17A analysis only involved the comparison of initial estimates to final estimates. 

The fourth DoD O&S cost study was published by the Government Accountability Office (GAO) in 
2010 [8]. This report initially sought to review O&S cost behavior of major defense programs, but ended 
up being more of an indictment of current Pentagon deficiencies with respect to tracking and managing 
O&S costs for its systems. Despite expressed reservations with the fidelity of the underlying data sources, 
the GAO did report O&S costing information for seven programs. For five of these systems, the analysis 
involved cost estimates only, but in two cases (the Air Force F-22A, and the Navy F/A-18E/F), estimates 
were compared to actuals, thereby enabling some insight into the accuracy of initial O&S cost estimates. 

The salient components of these O&S studies are summarized below in Table 1. Note that only subsets 
of the CNA report and the GAO report employed a methodology that allows meaningful characterization 
of O&S cost estimates. And since there were no numerical results in the CNA report regarding these 
systems, those few pages of the 2010 GAO report which analyze the F-22A and the F/A-18E/F would 
appear to represent the only published qualitative characterization of the accuracy of O&S cost estimates 
for DoD systems. 


Table 1. Summary of existing O&S cost studies on defense systems 


Source 

Year 

# of Systems 

Method 

Quant. Results 

OSD 

2009 

34 

“Cost Growth” in O&S Actuals 

n/a 

CNA 

2009 

23 

“Cost Growth” in O&S Estimates 

n/a 

CNA 

2009 

3 

O&S Estimates vs. O&S Actuals 

No 

IDA 

2010 

1 

“Cost Growth” in O&S Estimates 

n/a 

GAO 

2010 

5 

“Cost Growth” in O&S Estimates 

n/a 

GAO 

2010 

2 

O&S Estimates vs. O&S Actuals 

Yes 


2.2. O&S reporting 

Although the enactment of WSARA has undoubtedly served as a catalyst for increased focus on O&S 
cost issues in the DoD, it is also the case that an empirical analysis of O&S costs for defense systems 
would simply not have been possible until relatively recently. To conduct the analysis requires three 
elements: a valid source of predicted costs from the acquisition phase, a valid source of actual costs from 
the O&S phase, and enough elapsed time for a large number of programs to have accrued representative 
data from both phases. 

The obvious source for obtaining official estimates of O&S costs for major defense programs is the 
Selected Acquisition Report (SAR). SARs are required to be submitted at least annually for all major 
defense programs until they have been 90 percent acquired (further evidence of the imbalanced emphasis 


CNA justified this approach by noting they were unable to obtain true actual costs for the majority of the programs in the study, 
and did not wish to mix methodologies. 
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the DoD places on the acquisition phase) [9]. Starting in 1989, programs were required to include as part 
of each SAR a “full life-cycle cost analysis,” and soon thereafter (about 1990 in most cases), programs 
began providing estimates of Annual Unitized O&S cost (i.e., average O&S cost per unit, per year). 
Starting in 2001, most programs also began providing an estimate of Total O&S cost. 

Not long after O&S estimates were being required in the SARs, the DoD mandated that each military 
service maintain an historical database of actual O&S costs for its systems. The effort became known as 
Visibility’ and Management of Operating and Support Costs (VAMOSC), with each service component 
managing its own version. Though the primary focus of VAMOSC is on future planning and the 
development of O&S estimates, the nature of the database allows for actual O&S costs to be broken out 
by weapon system and by year [3,9]. Accordingly, VAMOSC data can serve as “ground truth” for actual 
system O&S costs, thus enabling an accuracy assessment of the O&S cost estimates found in the SARs. 

So with a consistent, time-phased, and reliable source for predicted O&S costs in one hand, and a 
viable source for obtaining actual O&S costs segregated by system in the other, all that remains is 
allowing enough time to pass for a sufficient amount of data to be collected for a valid comparison 
between the two cost figures. Since programs can take many years to develop and field, it is not 
unreasonable to suspect that it could take a couple of decades to amass sufficient data for a substantive 
analysis. In fact, the authors have screened all the data, and found that two decades (1991-2010) is enough 
time to obtain valid O&S cost estimates and actuals for over three dozen major defense programs. 

3. Proposed Methodology 

At this point, the basic methodology should be evident. Our first step is to annotate the predicted O&S 
cost estimate (or estimates in those cases where multiple measures of O&S cost are provided) from every 
SAR for a given system. We next use the VAMOSC data to establish the actual O&S costs for that 
system. Finally, we compare the actual O&S cost of the system to what it was predicted to be each year 
during its acquisition phase and characterize the accuracy of that estimate over time. The principle is 
simple enough, but there are a number of obstacles that complicate the proposed analytical methodology. 
In fact, multiple authoritative sources are dubious that such an analysis is feasible [8,11]. Consequently, 
the remainder of this paper is largely devoted to a discussion of how these obstacles can be overcome or 
mitigated. 

3.1. Methodological details 

The first challenge is one of system selection. There are hundreds of candidate programs that have 
submitted SARs; however, we have a rather stringent set of selection criteria. First, we eliminate any 
program that does not provide valid O&S cost estimates. This necessarily includes any program that 
stopped submitting SARs prior to about 1991 (recall that O&S cost estimates started first being reported 
around 1990). This also includes a surprising number of programs that either did not comply with the 
DoD requirement to provide O&S cost estimates, or that provided estimates that were clearly erroneous. 

Next, we remove programs for which we cannot obtain valid—and relatively stable—actual O&S 
costs. Obviously, this encompasses programs cancelled prior to becoming operational, but it also 
necessarily includes programs for which there are too few years of actual O&S cost data or too few units 
operationally deployed to have a reasonable expectation of having achieved steady state O&S costs. Also, 
similar to the difficulties encountered with the SARs, many programs must be eliminated because the 
VAMOSC data is unavailable or invalid. Another reason for excluding a program is that the level of cost 
reporting for the VAMOSC data is incommensurate with the SAR data; this is often the case with major 
system modifications, which may warrant SAR reporting, per se, but are not sufficiently delineated in 
VAMOSC with respect to the specific scope or phasing of the modification to allow for meaningful 
comparisons. 
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The next major challenge is reconciling discrepancies that arise in the cost element structure between 
the predicted costs in the SAR and the actual costs from VAMOSC. In some cases, VAMOSC is missing 
data for entire cost categories identified in the SAR. For instance, the Naval VAMOSC system is unable 
to allocate to specific weapon systems those expenditures in the category of “indirect costs” (e.g., 
personnel medical care and base services and support). In order to resolve the mismatch between 
predicted and actual O&S costs for this cost element, we normalize the Navy data by decrementing the 
amount of “indirect costs” from all the SAR estimates. In theory, this introduces an additional component 
of uncertainty into the analysis related to the relative accuracy of cost element categories; however, any 
ostensible effect is likely to be minimal as the missing VAMOSC costs represent, on average, less than 
ten percent of the projected O&S costs. 

For Navy systems, the issue of missing cost elements in the VAMOSC actuals can be readily 
mitigated, and, generally speaking, there is no analogous problem for the Air Force VAMOSC system. 
The Army VAMOSC data, however, is another matter. For Army programs, specific weapon system data 
is simply not allocated for most of the O&S cost categories, to include indirect costs, personnel costs, 
contractor costs, and sustaining support costs. This deficiency in the Army cost accounting systems is so 
significant that it precludes inclusion of any Army programs in this study (many of these deficiencies 
with the Army VAMOSC system are documented in the GAO report [8]). 

Another challenge related to this proposed methodology is the fact that, for many systems, the level of 
reporting between the estimates and the actuals is not commensurate. In some cases, a single O&S cost 
estimate is provided for the system, but actuals for that same system must be aggregated from multiple 
sources. This generally occurs for one of three reasons. One, it is a joint program and the VAMOSC cost 
data spans multiple service components. Two, the core mission of the system changes over its life such 
that costs are accrued via different databases. Three, system variants are combined in the SAR, but 
segregated in the VAMOSC systems. In all three cases, it is generally possible to consolidate the cost 
actuals such that estimates and actuals represent an “apples-to-apples” comparison. 

The complementary problem can also occur, whereby the estimates are broken out by variant (whether 
in a single SAR or multiple SARs), but the actuals are not segregated. This tends to pose a greater 
challenge as we must calculate a composite figure from the multiple O&S cost estimates in order to 
enable a valid comparison to the actuals. For these cases, we calculate the composite O&S cost estimate 
by weighting the relative number of units for each variant. This is a reasonably valid approach in most 
cases, but is arguably less so if the temporal phasing of the deployment of the variants is too great, or the 
actual relative proportion of the variants varies substantially from what was planned. In sum, while 
incommensurate treatment of system variants can certainly complicate the analysis, it can be accounted 
for, and it hardly represents a significant methodological barrier. 

There are two measures of O&S cost that can be assessed via this proposed methodology: Total O&S 
cost and Annual Unitized O&S cost 2 . Each measure—within the context of this study—has its own 
strengths and weaknesses. The Total O&S cost is a readily intuitive metric that offers (when summed 
with total acquisition cost) a direct means of establishing an estimate of system LCC, the most 
comprehensive and facile cost indicator for system value assessments. Although Total O&S costs are not 
explicitly stated for any system until 2001, we can sometimes infer an estimate for these earlier SARs. 
Note this is possible only if certain other information is provided in the SAR, such as the Annual Unitized 
O&S cost and the assumed operational service life. It should be noted that this inference does represent an 
overly simplistic view of Total O&S costs (e.g., it neglects, ramp-up/ramp down periods, attrition, 
refurbishments, etc.), but this is the basic method employed in the vast majority of later SARs to estimate 


2 A third metric is also theoretically possible: Operating cost per unit of time (i.e., cost per flight/steaming/driving hour). However, 
we elected to omit this metric for this study due to the relative paucity of predicted data points and the lack of inconsistency 
among programs in how precisely to employ this type of metric. 
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Total O&S costs. So while the inferential calculations we employ may be simplistic, they are at least 
consistent. 

There are, however, two notable problems in using Total O&S costs as a metric. One is that it is highly 
correlated to unit quantities. This can lead to particularly high degrees of estimate variance over time; or 
alternatively, it can provide a ready-made strategy for programs to effectively mask increasing estimates 
of Annual Unitized O&S costs, by reducing fielded quantities until Total O&S costs meet cost goals (this 
is the same gamesmanship strategy which has been curtailed on the acquisition side via the concept of the 
“unit cost breach"). The second problem with examining Total O&S costs is we cannot possibly know the 
true actual costs at this time. With two exceptions, none of the programs in this study has reached end of 
life, so the portion of Total O&S costs that has not yet been incurred must be estimated by prorating 
actual costs to date across the remainder of the expected operational service life, potentially creating 
another source of uncertainty. 

The Annual Unitized O&S cost may not provide direct insight into the system LCC, but the data tends 
to be more broadly available both on the estimate and actuals side. Estimates of Annual Unitized O&S 
cost often begin around 1991, and actual Annual Unitized O&S costs can be easily calculated for any year 
in which actual O&S costs are reported and the number of operational units can be determined. Of note, 
however, obtaining operational unit counts is not always as straightforward as one would expect. For 
aviation and maritime systems, unit counts are readily available in the VAMOSC databases, though in 
some cases—depending on the SAR assumptions—we use the available operational inventory counts vice 
the actual full inventory counts. For non-aviation and non-maritime systems, however, actual counts are 
not available in the VAMOSC databases. In these cases, alternative sources (e.g., the managing 
organizations) must be queried to obtain historical fielded quantities. Yet even with these unit count 
challenges, Annual Unitized O&S costs (as opposed to Total O&S costs) often provide a more valid 
comparative measure across similar contemporary systems or antecedent systems. Of course, in some 
cases, such as when the system consists of very few units or the system has a monolithic nature, the 
relevance of a unitized cost metric is diminished or lost. 

Perhaps surprisingly, inflation adjustment may represent the largest practical challenge with this 
proposed methodology. Clearly, to make valid comparisons between projected costs and actual costs, 
comparisons must be made in the same base year dollars. In theory, there are two approaches: either 
inflate the SAR estimates or deflate the VAMOSC actuals. In practice, the former option is not feasible, 
as the SAR cost categories do not cleanly map to specific inflation categories. And since the level of 
fidelity in the SAR O&S estimates does not allow insight into the specific inflation categories, it is not 
possible to accurately inflate these numbers based solely on the SAR data. Therefore, in all cases, we 
have chosen to deflate the VAMOSC data. This turns out to be a highly intricate task for most systems, as 
inflation factor adjustments can be quite abstruse and the specific calculations vary significantly based on 
service component and system type. Furthermore, since we cannot inflate the projected O&S costs, we 
cannot compare SAR estimates that are provided in different base years, even for the same program. This 
has the unfortunate effect of creating artificially separate analytical units (e.g., F-22 base year 1990 costs 
and F-22 base year 2005 costs), thus slightly diminishing our ability to compare systems and ascertain 
statistical patterns. 

There were many other minor problems involved with this methodology that are too subtle to address 
substantively in a paper of this length. Many of these smaller issues pertain to abnormalities discovered in 
the SAR O&S estimates of nearly every program. Examples include inconsistencies between tabular 
values and the narrative text, unstated assumptions, varying metric parameters, and incorrect units of 
measurement. In most cases, logical inferences were made and documented, such that the problematic 
estimate could be included in the study (though sometimes it was assigned a lower weighting in the 
analysis to reflect reduced confidence in the data). In other cases, however, the necessary inference could 
not be reasonably justified, in which case that particular SAR data was excluded from the analysis. 
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3.2. Methodological concerns 

There are a number of limitations and potential objections associated with the proposed methodology 
that warrant documenting or discussing more formally. Let’s begin with methodological constraints. 

The scope of applicability for this study is limited to major systems acquired by the Air Force or Navy. 
Only major defense programs produce SARs, and only the Air Force and Navy VAMOSC systems were 
able to provide actual O&S costs of sufficient fidelity to make comparisons to the SAR estimates valid. 
Moreover, this study is dominated (over 80 percent) by just two categories of systems: aviation and 
maritime. Of the systems that are not classified as aviation or maritime (i.e., space, communication, and 
munitions), it is certainly debatable whether standard O&S cost models can or should apply, thus 
potentially restricting the applicability of the findings further. If one does wish to make the case that these 
other types of systems should be omitted entirely from the analysis, it should be noted that this can be 
done with little impact on the statistical analysis, as there would still be a sufficient number of 
independent subjects to ensure that the central limit theorem remains valid. 

Another potential objection to this proposed methodology is the inherent bias toward programs that 
succeed. By the very nature of the O&S cost predictions, there is no way to vet their accuracy if the 
program is not fielded, i.e., if it does not experience O&S costs. And though our goal is to characterize the 
O&S cost estimates for all programs, the data only allows us to assess the consummated ones, which, of 
course, we do not know a priori. This presumably obvious point does have some subtle implications. One 
might hypothesize, for instance, that since a program’s fate is determined—at least in part—by its 
estimated life cycle costs, those programs that tend to underestimate their life cycle costs are more likely 
to survive into the O&S phase, thus biasing low our assessment of cost estimate accuracy. 

We also recognize there may be systemic biases in the actual O&S cost data. Not surprisingly, given 
the required phasing construct of this study, the vast majority of the O&S cost actuals (~85 percent) were 
accrued since 2001. And since the U.S. has been at war nearly that entire time, the O&S actuals may be 
distorted (though which systems, which direction, and to what degree are all open to debate). It is also the 
case that this past decade has been marked by extremely volatile (and record high) prices of oil, which is 
known to be a major driver of O&S costs. Finally, many of these programs have encountered extremely 
large quantity changes, which will necessarily create a disconnect between estimated and actual O&S 
costs, especially in the case of Total O&S costs. 

This brings us to a fundamental theoretical objection to the underlying premise of this study. It may be 
argued that it is inappropriate to compare actual O&S costs to estimated O&S costs when the assumptions 
used to construct the estimates are often fundamentally different from what occurred in reality. 
Uncertainty in oil prices, wartime vs. peacetime operations, and procurement quantity adjustments are just 
a subset of all the different sources of uncertainty that can generate large discrepancies between estimates 
and actuals. This precise criticism, in fact, is often expressed in the O&S literature, including the 
previously cited GAO report which cautions that “program changes complicate direct comparisons 
between estimated and actual costs [8].” One response to this concern is that comparison of estimates to 
actuals may be considered an inappropriate strategy for O&S costs only to the same extent that it is an 
inappropriate strategy for acquisition costs, which are still held accountable to an original baseline despite 
enormous (and frequently realized) sources of uncertainty. The only difference between the two cases is 
that, for O&S costs, the time horizon is longer and the dollars at stake are larger. 

4. Preliminary Results and Future Work 

Another response to the theoretical objection above is that we might mitigate the impact of 
mismatched costing parameters if we were to only compare initial and final—or the most recent—O&S 
cost estimates (the presumption being that by only considering estimates, the cost-parameter 
discrepancies are likely to be less). This tactic is enticing, and is, in fact, essentially the methodology used 
in the CNA study cited earlier. For this to be a valid approach, however, we have to assume the most 
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recent O&S cost estimate is a reasonably good proxy for actual O&S costs. Our initial results, however, 
do not support this assumption. We found that when considering the most recent estimates of total O&S 
cost, Air Force estimates were on average 10 percent lower than the actual costs and Navy estimates were 
on average 26 percent higher. It is not yet clear the reason for these rather surprising results; it may be the 
case that they can be best explained by the GAO observation that the services are routinely failing to 
update their O&S estimates based on the latest actuals [8], 

At this time, the data is still being analyzed using a variety of mixed model techniques. Preliminary 
analysis, however, has indicated very large—sometimes systemic—errors in the O&S cost estimates for 
Air Force and Navy programs that tend to change over time. Furthermore, we have discovered several 
intriguing correlations between the accuracy of a program’s O&S cost estimate and various program 
parameters, such as service component, system type, cost variance to date, and total acquisition cost. Of 
central interest to future work aimed at improving O&S cost estimating, the fact that various independent 
program variables such as these may have statistically significant relationships with the accuracy of O&S 
cost estimates suggests the possibility of developing a prediction model that accounts for the stochastic 
nature of the data. We intend to publish the full results of this study, including the feasibility of a 
stochastic cost prediction model, in a subsequent paper. 

5. Conclusion 

Given their tendency for downstream impacts, systemic absence of accountability, and continual lack 
of full funding, it is no wonder that sustainability considerations are often neglected, minimized, or traded 
off for near-term acquisition interests. Knowing the monumental impact of decisions made during the 
acquisition phase, there is even a sound argument based in systems engineering principles for doing so. 
But such an argument assumes that O&S costs can be ascertained with some degree of accuracy; else 
meaningful management and mitigation are not possible. Moreover, strategic decisions of program worth 
are often anchored in life cycle cost estimates, which simply cannot be validated if the reliability of the 
largest cost component is unknown. 

Therefore, it is critical that the DoD develop a means of assessing the accuracy of its O&S cost 
estimates. This paper presents a methodology for doing so, which allows for the time-series evaluation of 
over three dozen major defense programs. We expect the findings to provide insights into the current 
accuracy of O&S cost estimates as well as to provide a potential mechanism for improving those 
estimates. 
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